Stochastic Variance Reduction Methods for Saddle-Point Problems

We consider convex-concave saddle-point problems where the objective functionsmay be split in many components, and extend recent stochastic variance reductionmethods (such as SVRG or SAGA) to provide the first large-scale linearly conver-gent algorithms for this class of problems which are common in machine learning.While the algorithmic extension is straightforward, it comes with challenges andopportunities: (a) the convex minimization analysis does not apply and we usethe notion of monotone operators to prove convergence, showing in particular thatthe same algorithm applies to a larger class of problems, such as variational in-equalities, (b) there are two notions of splits, in terms of functions, or in terms ofpartial derivatives, (c) the split does need to be done with convex-concave terms,(d) non-uniform sampling is key to an efficient algorithm, both in theory and prac-tice, and (e) these incremental algorithms can be easily accelerated using a simpleextension of the “catalyst” framework, leading to an algorithm which is alwayssuperior to accelerated batch algorithms.